A prominent technique for self-supervised representation learning has been to contrast semantically similar and dissimilar pairs of samples. Without access to labels, dissimilar (negative) points are typically taken to be randomly sampled datapoints, implicitly accepting that these points may, in reality, actually have the same label. Perhaps unsurprisingly, we observe that sampling negative examples from truly different labels improves performance, in a synthetic setting where labels are available. Motivated by this observation, we develop a debiased contrastive objective that corrects for the sampling of same-label datapoints, even without knowledge of the true labels. Empirically, the proposed objective consistently outperforms the state-of-the-art for representation learning in vision, language, and reinforcement learning benchmarks. Theoretically, we establish generalization bounds for the downstream classification task.

A prominent technique for self-supervised representation learning has been to contrast semantically similar and dissimilar pairs of samples. Without access to labels, dissimilar (negative) points are typically taken to be randomly sampled datapoints, implicitly accepting that these points may, in reality, actually have the same label. Perhaps unsurprisingly, we observe that sampling negative examples from truly different labels improves performance, in a synthetic setting where labels are available. Motivated by this observation, we develop a debiased contrastive objective that corrects for the sampling of same-label datapoints, even without knowledge of the true labels. Empirically, the proposed objective consistently outperforms the state-of-the-art for representation learning in vision, language, and reinforcement learning benchmarks.

Weaknesses: The main weakness that I see with the paper is the mismatch between the theoretical analysis and the algorithm used in the experiments. The proposed estimator uses samples from the true "positive distribution" which consists of images from the same class. This is of course infeasible in a self-supervised setting where labels are unavailable. As a result, the authors approximate this distribution with the usual "positive distribution" which consists of random transformations of a single image. I understand that this two-step procedure is necessary to have an tractable analysis (using the true "positive distribution") and an experimental approach which is comparable to other self-supervised approaches (which use the approximate "positive distribution"), but the approximation of the "true positive" distribution by the other should be made explicit and discussed.

Reviewers unanimously agreed that this submission should be accepted -- it's well-written and proposes an interesting, well-motivated, and theoretically grounded idea. It's cheap and easy to implement and add to any existing contrastive learning approach, and the empirical evaluation showed strong results. Unsupervised contrastive learning is currently an area with plenty of interest among the NeurIPS audience so it is likely to be relevant/interesting to a large slice of the community. The authors' rebuttal addressed most of the reviewers' initial concerns. The paper could be further strengthened by evaluating the method in the full ImageNet-1K setting as this is the standard benchmark for most unsupervised visual representation learning work (even if the bias isn't large enough for the method to have much effect with 1K classes, this is useful for readers to know) -- the authors should strongly consider including these results in the camera-ready version of the paper.

A prominent technique for self-supervised representation learning has been to contrast semantically similar and dissimilar pairs of samples. Without access to labels, dissimilar (negative) points are typically taken to be randomly sampled datapoints, implicitly accepting that these points may, in reality, actually have the same label. Perhaps unsurprisingly, we observe that sampling negative examples from truly different labels improves performance, in a synthetic setting where labels are available. Motivated by this observation, we develop a debiased contrastive objective that corrects for the sampling of same-label datapoints, even without knowledge of the true labels. Empirically, the proposed objective consistently outperforms the state-of-the-art for representation learning in vision, language, and reinforcement learning benchmarks.

Self-supervised representation learning on image-text data facilitates crucial medical applications, such as image classification, visual grounding, and cross-modal retrieval. One common approach involves contrasting semantically similar (positive) and dissimilar (negative) pairs of data points. Drawing negative samples uniformly from the training data set introduces false negatives, i.e., samples that are treated as dissimilar but belong to the same class. In healthcare data, the underlying class distribution is nonuniform, implying that false negatives occur at a highly variable rate. To improve the quality of learned representations, we develop a novel approach that corrects for false negatives. Our method can be viewed as a variant of debiased contrastive learning that uses estimated sample-specific class probabilities. We provide theoretical analysis of the objective function and demonstrate the proposed approach on both image and paired image-text data sets. Our experiments illustrate empirical advantages of sample-specific debiasing.